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ABSTRACT 



The Project on State-Level Child Outcomes is designed to 
assist states and other groups to improve the measurement of child outcomes 
in state welfare evaluations and in other state data systems. This report 
summarizes the first meeting of the five states (Connecticut, Florida, 

Indiana, Iowa, Minnesota) that participated in the operational phase of the 
project concerning data collection related to child well-being and family 
processes. The primary purpose of the meeting was to come to a consensus on 
how to measure the common core constructs that had been previously selected 
by the project participants. This report details each state's updates on 
impacts and indicators studies. In addition, the report outlines the 
discussion on measurement of the constructs in the following areas: (1) 

income; (2) employment; (3) family formation; (4) stability and turbulence; 

(5) psychological well-being; (6) absent parent involvement; (7) use of heath 
and human services,* (8) consumption, percent of income spent on child care 
and rent; (9) child care,* (10) home environment and parenting practices,* (11) 
education; (12) health and safety,* and (13) social and emotional adjustment. 
The meeting summary continues with tips for training interviewers and 
monitoring data collection quality, lessons learned from other program 
evaluations regarding this area, and the technical assistance needs of states 
and evaluators for the operational phase of the project. The report concludes 
with a list of participants. An attachment to the report delineates the 
components of the core constructs. (KB) 
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Project Overview : The U.S. Department of Health and Human Services ' Office of Planning, Research, and 
Evaluation at the Administration for Children and Families (ACF), and Office of the Assistant Secretary for Planning 
and Evaluation (ASPE) are working together with states and other groups to improve the measurement of child 
outcomes in state welfare evaluations and in other state data systems. ACF is providing grants to states instituting 
welfare reform demonstrations to augment their demonstration evaluations with measures of child outcomes^and also 
to expand their data capability to track state-level indicators of child well-being on an ongoing basis. Under funding 
from ASPE and the other federal contributors and private foundations, the states are receiving technical support on 
these activities from leading researchers who are members of the NICHD Family and Child Well-Being Research 
Network. The Network’s technical support effort is led by Child Trends, Inc. 

The project has two phases. The first phase was a one-year planning and design phase which began October 1, 1996. 
The second operational phase is an implementation phase for data collection, analysis, and reporting activities which 
began October 1, 1997. Twelve states participated in the first phase: California, Connecticut, Florida, Illinois, 

Indiana, Iowa, Michigan, Minnesota, Ohio, Oregon, Vermont, and Virginia. Five states received funding for the 
operational phase. These include: Connecticut, Florida, Indiana, Iowa, and Minnesota. 

This report summarizes a meeting on implementing the common core of constructs of child well-being and family 
processes. This meeting was the first of the five states that are participating in the operational phase of the Project 
on State-Level Child Outcomes. 



The project is sponsored by ACF and ASPE. Additional federal funding to support this project has been provided by 
the U.S. Department of Agriculture, the National Institute of Child Health and Human Development (NICHD), and 
the Centers for Disease Control. Several private foundations have contributed funding to support the organization of 
national level meetings, the provision of technical assistance to the states, and the preparation and dissemination of 
written products. These include: the Annie E. Casey Foundation , the Edna McConnell Clark Foundation , the 
George Gund Foundation , and the Smith Richardson Foundation. 

The following individuals have been involved in multiple aspects of this project: 
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AlanYaffe Martha Moorehouse 

Ann Segal 
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Greg Duncan 
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MEETING OF THE PROJECT ON STATE-LEVEL CHILD OUTCOMES 

OPERATIONAL PHASE 
AGENDA 

DECEMBER 9, 1997 

CHILD TRENDS, INC. 

4301 Connecticut Avenue, NW Suite 100 
Washington, DC 20008 

9:00-9:30 CONTINENTAL BREAKFAST 

9:30-10:00 WELCOME AND OPENING REMARKS 

Howard Rolston, Office of Planning, Research, and Evaluation, ACF , U.S. DHHS 
Ann Segal and Martha Moore house, ASPE, U.S. DHHS 
Kristin Moore , President, Child Trends, Inc . 

1 0 :00- 11:00 STATE UPDATES ON IMPACTS AND INDICATORS STUDIES 

Connecticut 
Florida 
Indiana 
Iowa 

Minnesota 



1 1 :00- 1 2:30 MEASURING THE COMMON CORE OF CONSTRUCTS 

12:30-1:15 LUNCH 

1:15-2:30 MEASURING THE COMMON CORE OF CONSTRUCTS, Continued 



2:30-2:40 BREAK 

2:40-3 :40 INTERVIEWER TRAINING AND THE MONITORING OF DATA 

COLLECTION QUALITY 

► Questionnaires 

Lee Robeson, Response Analysis Corporation 

► Calendars 

Adria Gallup-Black, Manpower Demonstration Research Corporation 

► Interviewer Observations 
Martha Zaslow, Child Trends, Inc. 

► Audio Computer Assisted Self-Interviewing (CASI)/Self-administered Questionnaires (SAQs) 
Adria Gallup-Black, Manpower Demonstration Research Corporation 



3 :40-4: 1 5 LESSONS LEARNED FROM THE NATIONAL EVALUATION OF WELFARE TO WORK 

STRATEGIES (NEWS) ABOUT INTERVIEWER TRAINING AND THE MONITORING OF DATA 
COLLECTION QUALITY 

Lee Robeson, Response Analysis Corporation 



LESSONS LEARNED FROM FIELDING THE MINNESOTA FAMILY INVESTMENT (MFIP) 
PROGRAM QUESTIONNAIRE 

Adria Gallup-Black, Manpower Demonstration Research Corporation 

4:15-4:45 TECHNICAL ASSISTANCE NEEDS FOR THE OPERATIONAL 

PHASE 
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OVERVIEW OF THE OPERATIONAL PHASE 



The operational phase, which began October 1, 1997, is the second phase of the Project on State- 
Level Child Outcomes. Five states received funding for data collection, analysis, and reporting 
activities. These include: Connecticut, Florida, Indiana, Iowa, and Minnesota. This document 
summarizes the first working meeting of the operational phase, which was held on December 9, 
1997 at Child Trends, Inc., Washington, D.C. Meeting participants included representatives 
from the five operational phase states, their evaluators, federal government representatives, and 
the technical assistance team from Child Trends. The primary purpose of the meeting was to 
come to a consensus on how to measure the common core of constructs that was selected by the 
project participants during last year’s planning phase of the project. The second goal of the 
meeting was to provide information on interviewer training and the monitoring of data collection 
quality to carry forth the theme of commonality across the five states’ evaluations. 

I. WELCOME AND OPENING REMARKS 

Howard Rolston opened the meeting by emphasizing the importance of making this project a 
unified one. He said that developing a common set of constructs and measures is an important 
milestone. The critical end goal of the project is to create a data base that contains comparable 
measures. Rolston also indicated that he is confident that all five states will be able to field an 
in-home survey. HHS has prepared a prospectus that has been shared with several private 
foundations, and they also have had conversations with other federal government funders. The 
prospectus laid out several “add-ons” to the basic surveys. The highest priority is for all five 
states to do in-home surveys. The second highest priority is to build sample sizes. Next are 
linkages to administrative data bases. HHS also included in the prospectus as funding options a 
teacher survey and stand-alone studies of child care. Rolston said that the work that these 
projects are doing will serve as a tremendous legacy and long-term asset. 

Ann Segal noted that the government has a strong interest in getting some state-level information 
and developing a database. Part of the underlying commitment to the project, she noted, is that 
what happens to the children is an important part of welfare reform. 

Martha Moorehouse noted that these waiver evaluations build on the pioneering efforts of the 
JOBS evaluation (now called the National Evaluation of Welfare to Work Strategies or NEWS 
evaluation). This was the first attempt to embed a set of child outcomes measures in a major 
welfare evaluation focusing on adults, in order to track the effects of welfare reform on children. 
Getting rich information on children in a survey context is not always easy, but the NEWS study 
made considerable methodological progress and provides a history of what to do that will inform 
this project. The commonality in the current set of studies comes out of that history. 

Moorehouse also noted that it is important to look at the null hypothesis in this setting. We need 
to be sure that policies do no harm. Thus, both finding differences and also not finding 
differences in child outcomes is significant. She also noted the importance of taking a fine- 
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grained look at risk factors and emphasized the importance of understanding implementation. 
Common measures, common implementation, and common interviewer training are all essential. 
She commented that interviewer training is where commonality hits the road, that “the devil is 
really in the details.” She emphasized the seriousness of interviewer training and encouraged 
states to use NEWS as a model, with Child Trends’ technical help. 

Identifying herself as this project’s spokesperson for tracking indicators of child well-being over 
time, Moorehouse then encouraged states to look at administrative data, perhaps with 
experimental and control groups. She noted that the broader work of looking at indicators is not 
being led out of the welfare offices but indicated that the government still wants to support 
indicator efforts and is exploring other ways to support indicators of child well-being work as the 
operational phase of the impact studies continues. 

Kristin Moore highlighted two goals for the day. The first goal was to help state representatives 
develop and implement a useful data collection effort for their state. The second was to build a 
larger body of knowledge about how welfare policies affect children. She suggested that these 
two goals are generally compatible, and can be forwarded by a lot of sharing of knowledge and 
experience. Moore reiterated the theme for the day: Commonality. 

Moore noted that at previous meetings, state representatives had agreed on a set of common 
constructs, but had not yet selected specific measures to operationalize these constructs. Since 
Minnesota is already in the field, while other states will not be in the field for some time, this is a 
complex task, made easier by the availability of the Minnesota Family Independence Program 
(MFIP) questionnaire to use as a template. She commended Minnesota and Manpower 
Demonstration Research Corporation (MDRC) for their excellent work, noting that any 
subsequent comments be viewed not as criticism of their work but the need to use the MFIP 
instrument as a common reference point for seeking to implement the common core of 
constructs. Where possible, other states should attempt to replicate the MFIP questionnaire. 
Before moving on to discussions about specific measures, meeting attendees provided brief 
updates about each state. 

II. STATE UPDATES ON IMPACTS AND INDICATORS STUDIES 

Sharon McGroder from Child Trends introduced the state representatives. They were asked to 
discuss any changes in funding, the evaluation’s design, and project time tables. In addition, the 
representatives were asked to describe any recent changes in policies, and their state’s economic 
or political environments. Finally, they were asked to discuss any indicators work that their 
offices are conducting, as well as other indicators efforts being conducted in the state. 

1. CONNECTICUT 

Mark Heuschkel from Connecticut’s Department of Social Services began by discussing the 
status of their waiver evaluation. As a Track One state, they are maintaining their control group, 
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which is still subject to pre-TANF (i.e., AFDC) policies though, in a few cases, controls are 
subject to a few TANF provisions that are expected to affect very few families (e.g., making sure 
fleeing felons don’t get assistance). In terms of changes to their proposed “enhanced” child 
outcomes evaluation, Connecticut is now planning an 18-month “interim” survey to be fielded in 
Spring, 1998; this is in addition to the enhanced 36-month survey effort, which will commence in 
the Spring of 1 999. Some academic researchers in Connecticut and elsewhere are interested in 
collaborating on this. Connecticut is not currently funded by ACF to conduct an in-home survey 
or for administrative data collection, which hampers their ability to develop indicators under this 
project; they would like to track child welfare data over time. 

An additional study, funded state funds and including subjects from both within as well as 
outside the waiver evaluation research sites, will focus on tracking those who leave welfare due 
to time limits — which is already happening — and examine such characteristics as their financial 
well-being and living arrangements. Anecdotally, Heuschkel reported that about half of those are 
eligible for extensions, but many are not because they are working and have earnings above the 
TANF cash program’s payment standard. Heuschkel also reported, incidentally, that Connecticut 
has met their TANF participation rates for FY 1997 (mostly through subsidized employment). 

In terms of the economic and political context of Connecticut, the economy is starting on an 
upward turn. The unemployment rate is still lagging in the region. The national economy is 
“pulling up” Connecticut’s economy. 

Howard Rolston then asked about the inherent disincentive to increase the number of hours 
worked, given that Connecticut’s income disregards do not expire and are enough to allow 
recipients to become “comfortable” with the amount of money they get. He wanted to know if 
there is some effort to encourage people to increase work hours. Heuschkel noted an increased 
interest on the part of the state to address this issue, and there is a big effort to get people into 
higher-paying jobs. While addressing this “hours disincentive issue,” they are currently 
concentrating their efforts on people reaching the end of their time limits. 

2. FLORIDA 

Jeannee Elswick-Morrison from the Florida WAGES Office was “pinch-hitting” for the usual 
Florida representatives, Pat Hall and Don Winstead, who were attending another conference. 
Elswick-Morrison reported on several policy changes that have occurred in Florida. The Florida 
legislature has set a 48-month lifetime limit on welfare program participation. There is also a 
Work First Plus time limit. However, Florida is providing intensive case management for hard to 
place and long-term welfare recipients. They are piloting these programs in six counties. 

Among other changes taking place in Florida is the piloting of a privatization program for . 
welfare administration, including eligibility and termination. The legislature has required a 
privatization pilot program in three counties: one urban, one suburban, and one rural. These pilot 
programs will be up and running in the next two months. 
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Next, Elswick-Morrison gave an update on child care and child safety issues, including informal 
child care. Within the last three months, five children have died in Hillsborough county while 
they were in informal care arrangements. These events have precipitated changes in how 
informal child care can be arranged for welfare recipients. The legislature gave a directive,for 
the need to insure child safety in informal arrangements beyond the immediate family. This 
directive was interpreted as meaning if anyone beyond the maternal grandmother was'' used for 
informal care, there had to be a national background check prior to placement. However, 
interpretation of informal care placement approval was not coordinated with welfare reform prior 
to implementation, and much confusion has resulted. (In Florida, the State Department is the 
child care provider, with placement carried out by private, non-profits; welfare reform, however, 
is operated under local WAGES coalitions. Elswick-Morrison noted that often one hand doesn’t 
know what the other is doing.) Currently, Florida is working on getting the new child care policy 
in place and getting it well-known across the state. 

Also, the Florida legislature is meeting and considering issues related to work requirements. 

They met the work requirement for FY 1997. They are changing what is an allowable work 
activity, and are also dealing with the issue of transportation (which is a big issue for the state of 
Florida, since there is no public transportation). Elswick-Morrison also mentioned that they are 
trying to retain control of some dollars that were slated for welfare and now seem to be up for 
grabs. Florida is now entering the tourist season; there is much seasonal employment in the state. 
Elswick-Morrison reviewed how welfare reform is handled in Florida. She likened it to a tripod, 
where the major players in welfare reform are the Department of Labor and Employment 
Security, the Department of Children and Families, and local WAGES coalitions. 

For a discussion of indicators, Caroline Herrington of Florida State University took the floor. 
Herrington reported that a state-wide task force consisting of state agencies (Department of 
Labor, Department of Education, etc.), oversight groups, Management Information Systems 
people, and university people (who are conducting research or who have expertise) had been 
convened and was in the process of compiling a set of core indicators specific to Florida. She 
mentioned that this list of indicators would hopefully be comparable to the Project on State-Level 
Child Outcomes’ common core of constructs. This process should be finished in the next month 
or so. 

3. INDIANA 

Tracey Nixon provided the update on Indiana’s activities. There have been several recent policy 
changes in the state. In June of 1997, Indiana eliminated a two-track system in which one track 
was not subjected to the 24-month time limit until it reached a higher track. As of June, there is 
only one track and all adults are subject to the 24-month time limit. Changes are also being made 
to rules for exemptions from the time limit. Currently, parents with children age one year or 
younger are exempt. This age limit will be reduced in stages, until by December 1998, only 
parents who are responsible for the care of children 12 weeks of age or younger will be exempt. 
Two requirements were added to the personal responsibility agreement that program participants 
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must sign. These include: agreeing to raise children in a drug-free home, and agreeing to raise 
children in a home free from domestic violence. Nixon reported that the number of people who 
are sanctioned has actually declined. She attributed part of this decline to the fact that 
employment and training contractors have become more proactive in getting people to work. 

With regard to the impacts evaluation, Indiana has taken several steps to strengthen the 
experiment. One difficulty that they were experiencing is that caseworkers in the entire state 
needed to be trained in both the new (i.e., TANF/waivers) and old (AFDC) policies. Beginning 
in January, 1998, random assignment will be limited to 12 counties. Each office will have a 
designated “control group” caseworker who will be very familiar with the AFDC policies. The 
other difficulty that the state has had with the experiment is that there was a period of several 
months during which people were not being assigned to the control group because of a computer 
problem. Nixon expects that control group assignment will be reestablished in January, 1998. 
Indiana will field a total of four surveys. The first has already been fielded, the next will be 
fielded in early 1999, the child well-being survey will be fielded in 2000, and the fourth will be 
fielded in 2001. Abt Associates is analyzing the first survey results and has completed an 
indicators study. 

4. IOWA 

Deb Bingaman described the state’s recent policy changes and the context in which their 
evaluation is being conducted. Bingaman said that welfare reform in her state has nearly 
remained the same since 1 993. At this point, there is some “tweaking” of the policies. Iowa has 
a five-year time limit for receipt of assistance. They have increased incentives and have found 
that almost all of the state’s caseload are participating in employment or training. Only those 
who are disabled are excluded from employment or training'. They are currently conducting 
several small pilot studies on systemic barriers such as transportation and child care. They will 
provide recommendations to the state legislature by January, 1988 on whether funding should be 
increased. 

Bingaman also noted that the state’s sanctions only last for six months. Based on results from 
the Limited Benefit survey, they have concluded that families are not suffering extreme 
deprivation or hardship. Also, Mathematica Policy Research has completed an interim impacts 
study report. This report is currently under review by Bingaman’ s department. 

With regard to indicators, Bingaman said that her department has very little authority to initiate 
projects in this area. However, several indicators activities are happening in the state. The 
Council on Human Investment is doing indicators work in the child abuse and neglect and foster 
care areas. In addition, the Lieutenant governor’s department is restructuring a task force which 
will recommend a set of indicators for children ages 0 to 5. 

Christine Ross from Mathematica Policy Research described the state’s impacts evaluation. She 
said that Iowa will be the second state in the field. They plan to field both the Track 1 survey and 
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the child impact survey by the end of January, 1998. The Track 1 survey will be a 30-minute 
telephone survey of approximately 3,000 people. The child impact survey will be in-home and 
will include about 2,000 families. The families that overlap between the two surveys will be 
contacted first by phone for the Track 1 survey. The child impact survey will be schedule^ for a 
later point in time. There will be some overlap between the Track 1 and common core 
constructs. Ross said that they will have to set some priorities for constructs in both the Track 1 
and child impact surveys. 

5. MINNESOTA 

Joel Kvamme from Minnesota and Adria Gallup-Black from Manpower Demonstration Research 
Corporation described Minnesota’s policy environment and evaluation activities. Kvamme said 
that the demonstration program that Minnesota has had in place since 1 994 will be the state’s 
TANF program. Beginning in January 1998, the Minnesota Family Investment Program (MFIP) 
will be implemented state-wide. There has been a change of focus in the state to take a broader 
look at working poor families. The 1997 legislature increased the state’s investments in child 
care and health care. Kvamme also said that there has been a lot of attention focused on what is 
happening to children in the state. 

With regard to the adult impacts evaluation, Gallup-Black reported that MDRC has completed an 
1 8-month follow-up report that was done in three urban counties, 12 months after random 
assignment. They found positive employment and earnings impacts, and greater welfare receipt, 
but a reduction in poverty. The child impacts evaluation has gone forward in seven counties and 
the control groups have remained intact. The MFIP child outcomes questionnaire went into the 
field in August of 1997 and will be out of the field by March, 1998. They are expecting 
approximately 2,100 child completions, and 1,500 non-child survey completions. 

III. MEASURING THE COMMON CORE OF CONSTRUCTS - PART I 

Kristin Moore began by directing attendees to the one-page “constructs” sheet (see Attachment 
A), which lists all the common core constructs agreed upon during the 12-state planning phase. 
She said that for each construct, a member of the Child Trends staff will present the rationale for 
why it is important to gather information on the construct, how it is measured in the MFIP 
questionnaire, how the technical assistance team comprised of members of the NICHD Network 
proposed measuring it in the measures notebook for core constructs, and Child Trends’ current 
recommendation, which recognizes the need to balance consistency across states with the best 
possible measure. 

1. INCOME 

Kristin Moore led the discussion on constructs in the income domain. By altering labor-force 
behavior, the size and duration of cash grants, and family composition, numerous waiver 
provisions may produce important changes in the level of family income. Family income can 
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affect children because money can be used to purchase things that are essential for growth and 
development, such as food, quality child care, activities, and lessons, and other resources. Lack 
of resources may mean that low-income children do not have as many stimulating experiences as 
do children in more affluent families. Money may also provide for greater safety and stability. A 
continual struggle for resources may lead children in low-income homes to experience less 
predictable daily routines and more residential moves than children in higher income homes. 

Last year’s planning phase project participants selected the following income-related constructs 
for the common core: total income; sources of income; stability of income; and financial 
strain/material hardship. 

la. Total income. While noting the value of MDRC’s decisions on measuring monthly income 
in the MFIP questionnaire for assessing adult outcomes, Moore noted that, for purposes of child 
outcomes, income needs to be measured over a longer time period, e.g., annually, because child 
well-being is affected by longer-term economic well-being. Child Trends recommended that 
other states use the two CPS items as proposed in the “Revised Measures Notebook for Core 
Constructs.” A discussion followed focusing on: (1) whether annual income can be measured 
reliably; (2) technical issues surrounding measurement of annual income, and (3) options for how 
best to measure annual income. Below is a summary of the discussion of these three issues. 

First, Barbara Goldman (MDRC) questioned whether reliable annual income data can be 
collected in a survey, which is why they are focusing on obtaining monthly information from the 
survey and from administrative records. David Fein (Abt) noted that when “lining up” survey 
and administrative reports of annual income, there is wide variability in measures. Barbara 
Goldman concurred, reporting that 70-75 percent of surveyed income is accounted for by 
administrative data on income. Christine Ross (MPR) proposed using administrative data to 
create an annual amount since it may be more accurate than surveyed household income. 
However, Howard Rolston noted that food stamps are not taxable and, thus, would not appear on 
tax records. Following this line of reasoning, Barbara Goldman asked, “What else can’t we get 
from administrative records?” Child support was mentioned. 

However, Moore noted that there may be good reasons why these two numbers might not be 
equal; for example, to the extent that recipients earn money under the table (e.g., babysitting) 
which is not reported in administrative records, such non-reported income is not captured in 
administrative data but may be captured in surveys. In addition, she argued that the two CPS 
items work well according to Greg Duncan (an economist and NICHD Network member at 
Northwestern University). George Cave (Child Trends) proposed that the issue is whether you 
can go from monthly income to poverty status, which is based on annual income and which is an 
important construct for relating to child outcomes. He suggested that asking summary questions 
such as the two CPS items would not take much time and would allow a poverty level to be 
calculated (along with information on family size). Howard concurred that it could be obtained 
quickly with these two items, to which Moore added that it could serve as a “signal,” even if it is 
not precisely correct. She noted that annual income will not only be used in impact analyses; it 
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will also be used in non-experimental analyses of child well-being as an “intervening 
mechanism.” 

Barbara Goldman suggested pretesting these two CPS items on the next survey (Iowa’s), and 
recording $10,000 intervals, though others said that $10,000 intervals would not provide a 
sufficiently detailed income distribution. David Fein concurred that pretesting would 
demonstrate not only how valid the measure is but also the degree of bias and, most importantly, 
whether the bias is the same for the experimental and control group respondents— in which case 
impact estimates will not be affected. 

In terms of technical issues, Lee Robeson asked whether family or household income is 
preferred. Moore responded that household income is preferred. Robeson then noted that 
welfare recipients do not think in terms of annual income, which exacerbates the problem of 
getting a reliable estimate from a survey. Jeannee Elswick-Morrison (FL) concurred, saying that 
sanctioning requires respondents to adjust the amount on a monthly basis. Mark Heuschkel (CT) 
noted that characterizing changes in income due to sanctioning, time limits, and other 
fluctuations makes it hard for clients to accurately report. Barbara Goldman expressed concern 
that the amount of time it would take to get a reliable survey estimate of annual income may not 
be worth it. 

Given monthly fluctuations in income, Jeannee Elswick-Morrison (FL) suggested that if 
respondents do not know their annual income, maybe they can provide an educated guess on a 
range of income. Several options were discussed, such as asking respondents if “this past month 
was typical?,” using categories of ranges of annual income, or following up the monthly income 
questions with “if we multiply this amount by 12, is this an accurate estimate of your annual 
income?” The group agreed to use a follow-up question listing categories of ranges of annual 
income. Child Trends will provide states and their evaluators with items to pre-test. As items 
are finalized (presumably for Iowa), final items will be forwarded. In writing these items, Fein 
recommended attention to four technical issues: (1) which 12 months are typical? (2) what about 
food stamps? (3) before tax or after tax income? and (4) family or household income? Moore 
reiterated that household income is preferred; it is easier to define a household than a family. 

lb. Material Hardship. Moore noted that the measure of material hardship included in the 
MFIP questionnaire was the same one that was recommended by Child Trends. The group 
agreed to include the same measure. 

Child Trends also recommended getting permission to obtain administrative record data in the 
future if required by the state. One of the evaluators pointed out that in welfare evaluations, 
permission is not needed. However, others said that parental consent is necessary to obtain 
school records data. States were urged to obtain parental consent just in case the teacher surveys 
are funded; an example of how this is done is on page 103 of the MFIP questionnaire. 
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2. EMPLOYMENT 



Kristin Moore led the discussion on constructs in the employment domain. How changes in the 
number and timing of work hours of adult women and other family members affect family life 
and children is a matter of considerable debate. On the one hand, employment may reduce time 
for the care of children, for monitoring the activities and behavior of older children,'- and for 
carrying out household responsibilities. On the other hand, employment and increased income 
may enhance parents' sense of financial security, self-worth, efficacy, and status. Children’s 
activities and time use are very likely to be affected by changes in maternal employment. 

Further, children observing their parents succeeding in employment may lead to increases in 
children's aspirations and optimism about the future. Yet if parents experience a substantial 
increase in stress in association with their employment, and/or experience job insecurity and 
instability, this could have negative implications both for the quality of parent-child relations and 
for the children’s aspirations. The constructs that the 12 states selected in this domain include: 
any vs. no employment; health benefits through employment; wages; hours of employment; 
stability of employment; education/licenses; hard job skills; multiple jobs concurrently; and 
barriers to employment. 

2a. Employment History and Job Characteristics. Kristin Moore began the discussion on 
employment by commending MDRC for obtaining detailed employment information in the MFIP 
questionnaire, but wondered whether the child care calendar was used to help respondents 
remember jobs. Adria Gallup^Black said that the reverse is true: jobs appear on the child care 
calendar as an “anchor” to help the respondent remember child care arrangements. Barbara 
Goldman noted that low-income populations are more apt to have shift work employment, and 
are less likely to know their schedule (which days, what hours) weeks in advance. She 
recommended getting this information at least for the most recent job; she also suggested 
obtaining the reason for job loss. (MDRC is putting this back into the Florida and Connecticut 
surveys). Deb Bingaman (IA) noted that their survey (to be fielded in January, 1998), is already 
asking how a job ended and why). Mathematica Policy Research will share these items with the 
project team. 

Christine Ross asked how far back an employment history should be obtained. She suggested 
obtaining the last two years. Moore noted that employment since random assignment and since 
the last survey date are other possibilities. This is an “absolute versus relative time” issue. She 
agreed that a five year employment history would be difficult, but also noted that it was done in 
the National Evaluation of Welfare to Work Strategies Child Outcomes Study on the child care 
calendar which covered the target child’s first five years of life. However, Christine Ross noted 
that this sample is 5- to 12-year-olds, and getting a history since birth would mean more than five 
years for most sample children. Adria Gallup-Black noted that in MFIP, they are getting detailed 
information only about the most recent job, although a job history is collected for the entire three 
year period since random assignment. Howard Rolston suggested counting no more than the 
three most recent jobs; he also noted that going back only two years may not capture the 
respondent’s first job off welfare, but we may have to live with that. Moore noted that while 
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welfare officials may want to know about the first job off welfare, for child well-being, the most 
recent jobs are more salient. Moore proposed that as part of the common core, states obtain a job 
history for the last two years, with the option of collecting more years if they want to. Barbara 
Goldman recommended getting information about (1) reasons for job loss, (2) shift work, .(3) 
rotating hours for each job in last year (see National Evaluation of Welfare to Work Strategies 
Five Year survey), and (4) an employment calendar. Howard Rolston noted that it wbuld be nice 
to have each of these in each site, since many states are doing it anyway (except MFIP), 
especially as they relate to child outcomes. The group reached consensus to get reasons for job 
loss, shift work, rotating hours for each job in the last year, and the job history for at least the last 
two years. 

Coordinating with the child care questions, Christine Ross noted that Iowa is planning to obtain 
current child care arrangements for all 5- to 12-year-old children in the family/household and 
obtain a detailed history for 5- to 7-year-olds. Dave Fein wondered whether this would focus 
only on child care while the mother was employed. Martha Zaslow noted that there is lots of 
child care used by mothers who are not employed, as they may be in training, so employment 
should not be tied into the child care calendar. This issue was tabled until the discussion on child 
care. 

2b. Hard skills. Kristin Moore noted that, beyond asking about educational attainment and 
licensing, there were not measures of “hard skills” in the MFIP questionnaire. The group 
discussed including additional items specifically pertaining to human capital and 
“promotionability.” However, there was consensus that these additional questions would be 
optional, not part of the common core. 

3. FAMILY FORMATION 

Kristin Moore led the discussion on constructs in the family formation domain. One of the stated 
goals of welfare reform on the national level is to reduce non-marital births and to end potential 
incentives that exist for family dissolution. While most waiver policies do not directly seek to 
influence what kind of families recipients live in, the policies may have consequences which 
affect family formation and dissolution. The following constructs were selected for inclusion in 
the common core: nonmarital/marital birth; child/family living arrangements; marital status of 
mother and whether she is or was married to the biological or non-biological father. (Note that a 
discussion about child living arrangements is covered in the stability/turbulence section). 

3a. Nonmarital birth. Kristin Moore noted that, while obtaining information about additional 
births since random assignment and about whether the mother had married since random 
assignment, MFIP does not measure whether any additional births since random assignment 
occurred when the mother was married to the biological father. She noted that Congress was not 
only interested in tracking non-marital births but also whether births occurred in the context of a 
two-parent biological family. Martha Moorehouse then asked if we would have child outcomes 
on the nonfocal child who was bom to biological parents since random assignment. Howard 
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Rolston noted that this is not a central issue for this particular study, and the group concluded this 
would not be part of the core, but states can pursue this on their own. 

IV. MEASURING THE COMMON CORE OF CONSTRUCTS - PART II 

4. STABILITY AND TURBULENCE 

Suzanne Miller Le Menestrel led the discussion on measures for the Stability and Turbulence 
domain. During last year’s planning phase, the project participants voted on including several 
aspects of stability and turbulence for the common core. These include: foster care placements; 
whether the child was not or is not living with his or her family; stability in child care; stability in 
income; number of moves of residence; and changes in marital status or cohabitation. 

Turbulence means that a child and family is experiencing multiple changes in life circumstances. 
It is possible that some welfare policies may bring about changes in several areas of a child’s 
family life, such as changes in child care, place of residence, or family income. Some waiver 
provisions might provide supports that may make children’s lives more stable, whereas others 
might increase turbulence in children’s lives. Turbulence during childhood in terms of family 
structure and living arrangements, family relocation, schools attended, child care, and income 
level, has been shown to be associated with a number of negative outcomes for children and 
young adults. 

4a. Foster care and why the child is not living with his or her family. These constructs are 
being measured in MFIP by asking parents whether they and their child have been separated 
since random assignment and how many times this has happened. For the most recent separation, 
the parent is asked the main reason why the separation occurred. Placement in foster care can be 
captured in this follow-up question. 

4b. Stability in child care. This is measured in the MFIP questionnaire using several different 
types of questions. There is a series of questions about the types of arrangements that the parent 
has used since random assignment and how many different arrangements of that type the child 
attended for a month or more. Second, using the calendar, one can ascertain the number of 
different child care arrangements that were used for a month or more since random assignment. 
For the most recent arrangement (that is, the one used in the last full week), parents are asked 
how many caretakers a child has, how many children are in the arrangement, and the number of 
hours the child spent in the arrangement. Stability in child care was discussed more fully in the 
section on child care. 

4c. Stability in income. This construct is more difficult to measure. In the MFIP questionnaire, 
data on income sources are only collected for the prior month. Thus, a variable which measures 
stability in income will need to be created from data collected in prior surveys and via 
administrative records since random assignment such as earnings from UI records, AFDC and 
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food stamp payments. In order to measure stability in income since random assignment, states 
decided to use administrative records data. 

4d. Changes in residence. This is being measured in MFIP by asking respondents a question 
from the National Evaluation of Welfare to Work Strategies Five Year survey which asks how 
many times they have moved since random assignment. - 

4e. Changes in marital status and cohabitation. These are being measured in MFIP using a 
series of questions from the New Hope evaluation. Respondents are asked if they got married, 
separated or divorced, or started or stopped living with someone since random assignment, and if 
these events occurred, how many times. 

The most important decision point was to determine a time period in which to construct a 
composite measure of stability or a composite measure of turbulence. Because the constructs 
that were selected for the common core are being measured since random assignment in the 
MFIP questionnaire, it would be possible to get a picture of stability and turbulence for this time 
period. With some of the questions included in MFIP, such as the question about recent 
separations, it would not be possible to create a stability and turbulence composite measure for a 
time period such as the last year or month, because the specific dates for the separations cannot 
be ascertained. 

The meeting participants decided that it would be important to find out whether the focal child 
was ever in foster care over some period of time, such as in the past two years. Project 
participants agreed to use the MFIP items to measure number of changes in residence and in 
marital status and cohabitation. Finally, upon Child Trends’ recommendation, everyone agreed to 
add to the common core one item from the National Survey of America’s Families which asks 
about the number of school changes the focal child has experienced in the past two years. 

5. PSYCHOLOGICAL WELL-BEING 

Kathryn Tout led the discussion about psychological well-being. Maternal depression was the 
aspect of psychological well-being that the group focused on in the common core. Depression is 
defined as a “negative mood state so extreme that it interferes with daily functioning and 
productive activity.” There are two general ways in which program participation could influence 
maternal depression. On one hand, mothers who are employed in low- wage jobs with few 
benefits or who hit time limits and sanctions may experience more depression. We know from 
prior research that depression is related to diminished parenting skills, high levels of child 
behavior problems and school difficulties, and poor physical health for children. On the other 
hand, mothers who secure employment and become self-sufficient may show decreased levels of 
depression and, in turn, may have children who show fewer negative developmental outcomes. 

In the common core, Child Trends recommended using the Center for Epidemiological Studies 
Depression Scale which is a 20-item questionnaire that has been used in other welfare program 
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evaluations and which has high reliability and validity. MFIP adopted the full scale (p. 90) and 
states agreed to follow suit. 

6. ABSENT PARENT INVOLVEMENT 

Suzanne Le Menestrel led the discussion about the measurement of constructs in the absent 
parent involvement domain. Absent parent involvement was a domain of interest to the states 
because many of the waiver provisions are aimed at increasing the role of the father in his child’s 
life, mainly through paternity establishment, enforcement of child support collection, and 
allowing families to keep more or all of the child support awards by passing through the monies. 
A possible consequence of these provisions is that fathers may see their children more frequently 
and the relationships between fathers and their children may be enhanced. On the other hand, 
forcing fathers to acknowledge their paternity and pay child support might increase feelings of 
anger and resentment toward both the mother and the child. Prior studies provide evidence that 
provision of child support is related to several areas of child well-being including cognitive 
development, academic achievement, and behavior problems. The planning phase project 
participants decided to focus on the following aspects of absent parent involvement as part of the 
common core: whether child support is or was provided; whether paternity was established; and 
the frequency of contact that the father has with the target child. 

6a. Whether child support is provided. This is measured in MFIP with a series of questions 
which ask if the child is covered by a child support order, whether any money was received from 
the child support office or from the father directly in the past year, and whether the father has 
provided informal noncash support such as buying gifts for the child, babysitting, or caring for 
him or her overnight. Jeannee Elswick-Morrision from Florida asked whether these questions 
could measure whether child support was not provided (or accepted by the mother) because it is 
actually in the child’s best interest not to see his or her father or receive support from him 
because of abuse or other reasons. It was agreed that the states could add questions which could 
answer that question, if they were interested in reasons why child support was not provided. 

6b. Whether paternity was established. This construct is measured in MFIP by asking parents 
if papers have been filed with a court or legal agency that establish the father as the legal father. 
There is also a question which asks whether the respondent was married to the child’s father at 
the time the child was bom. These questions were accepted for inclusion in the common core. 

6c. The frequency of contact that the absent parent has with the focal child. This is 
measured using two questions asking if the father has talked to the child on the phone or sent him 
a letter or card. Child Trends recommended adding an additional question about “face to face” 
contact that the child has had with his natural father in the past 12 months. This recommendation 
was supported by all project participants. 
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7. CONSUMPTION 



Kathryn Tout led the discussion about consumption, which refers to patterns of family resource 
use or the decisions families make to allocate scarce resources for their children. Last year’s 
project participants had decided to measure consumption in the common core as the percent of 
income spent on child care and housing. Program participation can affect consumption patterns 
by increasing or decreasing income. For example, if family income increases and a smaller 
percentage of income needs to be spent on housing and child care, more income is left over for 
other purposes. If family income decreases, families may need to move to poorer quality housing 
or choose lower quality child care settings. These patterns of consumption directly affect the 
contexts, such as neighborhoods and child care, in which children spend their days and thus are 
important for child well-being. 

In the common core, Child Trends recommended using questions that ask families how much is 
spent per week or per month on housing (including rent or mortgage, insurance and taxes) and 
child care. 

MFIP included questions assessing percent of income spent on both housing (p. 68, question 
AA4a) and child care (p. 58, question H9) in its Basic Survey. Child Trends recommended that 
the other states include the same questions. 

Jeannee Elswick-Morrision asked if the impact of subsidies is captured in a measure of 
consumption (e.g., if a family only pays $4/week for child care)? Howard Rolston replied that 
the importance of a consumption measure comes from assessing what the family pays. Thus, if a 
family pays only $4/week (and the rest is covered by subsidies), then the $4 figure is what is 
recorded. 

8. USE OF HEALTH AND HUMAN SERVICES 

Tamara F-Ialle led the discussion for this domain. Halle noted that access to health and human 
services is likely to benefit children indirectly, by reducing parental work disincentives, and 
directly, by providing services that will promote children’s overall health, and educational and 
socioemotional development. The recommendations made in the common core of constructs for 
Health and Human Services include specific questions about food stamps, medical insurance 
coverage for the parent and child, child care subsidies, and access to medical care. 

8a. Food stamps. MFIP included questions about food stamps from the National Evaluation of 
Welfare to Work Strategies Five Year survey (Section.F, page 38; A5, page 3). Howard Rolston 
urged states to get food stamp data from administrative data rather than adding questions to their 
surveys. Barbara Goldman explained that some of the food stamp questions in MFIP were there 
because MFIP did a cash out. MFIP also included an additional question about receiving MFIP 
benefits, net of other food assistance (F4, page 39). It was agreed that states would try to get 
food stamp information from administrative data; if they were unable, they would use the MFIP 
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items. Adria Gallup-Black cautioned that there may be a problem with using administrative data 
if more than one food stamp recipient unit resides in a household. 

8b. Medicaid/Medical Insurance. MFIP used a series of questions from the New Hope-survey 
to ask about insurance coverage in the past month (Section I, page 60). In addition, two items in 
the Material Hardship Scale (2Jf & 2Jg, page 63) ask if anyone in the household postponed 
medical or dental care. It was agreed that other states would use the MFIP items. States were 
also urged to consider (as an option) getting additional information for reasons why medical care 
was postponed (question F17 from National Survey of America’s Families was recommended). 

8c. Child care subsidies. Questions about child care subsidies were covered in Section H of the 
MFIP survey (H3 & H7, pages 55-57). It was agreed that states would use the MFIP items 
(although other states will substitute their own state programs where MFIP is explicitly 
mentioned). 

8d. Access to medical care. MFIP has questions about source and regularity of dental and 
medical care for the child (Section CC, page 78). States agreed to use these items. 

9. CHILD CARE 

Martha Zaslow led the discussion on the measurement of constructs in the child care domain. 

The work requirements in place with many of the waivers and with TANF are likely to affect the 
use of child care. There is evidence that child care can affect the mother’s ability to get and 
maintain a job. The use of child care by respondents is also likely to affect the development of 
the children. In particular, the research literature suggests that high quality early childhood 
programs can play a significant, positive role in the lives of disadvantaged children. Zaslow 
noted that the recommendation for the common core regarding child care measures had been to 
address extent of care used for the focal child, types of care used; quality of care used; and 
stability of care. (Other sections of the common core address issues concerning paying for child 
care). 

These aspects of child care are well covered in MFIP, particularly in section HH. Very little is 
missing in terms of particular items that had been recommended for the core (see below). In 
addition, MFIP goes beyond the core recommendations in the sense that it includes the full series 
of questions developed by Emlen concerning reactions to child care (e.g., “CHILD feels safe and 
secure” in the primary arrangement; with mother responding never, sometimes, often or always). 
This set of questions had been noted as optional in the core recommendations. 

9a. Type, Extent, Quality, Stability. Zaslow noted that the MFIP survey covers child care in 
two very different ways. Section H asks about child care specifically for the mother ’s current or 
most recent job and regarding all of the mother 's children. By contrast, Section DD asks about 
the entire period of time since random assignment, about child care used for any reason, not only 
for employment, and about the focal child only. Zaslow noted that the differences in approach in 
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asking about child care in these two sections are important. For example, regarding the fact that 
one section focuses on child care for purposes of employment but the other does not require 
employment, it is important to note that national data indicate that child care occurs for reasons 
other than employment. In the context of welfare waiver evaluations, one reason for using care 
other than employment might be for participation in education or training activities or unpaid 
work. 

Zaslow reviewed the content of these two child care sections in MFIP: Section H asks where the 
care is occurring (or occurred); how many of the mother’s children are in the care situation; 
whether the care giver is paid and who pays for all or most of the care; which particular types of 
care are used for any of the mother’s children and for how many hours per week; the total 
payment for child care for all of the mother’s children; and whether there is assistance with 
paying. Section DD asks, for the focal child, whether particular types of care have been used 
since random assignment; how many different persons or arrangements of each type have been 
used since random assignment; and whether the care giver for each type of care is younger than 
13, between 13 and 17, or older. 

One item that had been recommended for the core was not included in MFIP: “As far as you 
know, is your child care provider licensed or regulated by the state?” (Yes/No/Don’t Know; This 
question was to be asked regarding in-home child care but not center care). The group decided 
that this should be included among the items used in common where this is still possible. 

There is at present no item concerning accidents or injuries while the focal child is in child care. 
The morning discussion, especially comments made by the representatives from Florida, had 
raised the possibility that such accidents or injuries might occur. The group decided to add a 
question to the child care section asking whether since random assignment the focal child had 
experienced an accident or injury while in child care that resulted in a visit to the emergency 
room. The Child Trends team will make this item available to the state teams. 

Ross noted an alternative to asking about care quality that might perhaps be more fitting to a 
sample of school age children: asking about the three activities that the child engages in most 
while in the primary child care arrangement. Such an item would reveal, for example, whether 
watching television is the most frequent activity. Ross noted that they will be pretesting this item 
shortly for the Iowa evaluation. The group decided that this item should be optional. The Child 
Trends team will make this item available to all the state teams and provide any information 
provided by Mathematica Policy Research from the pretest. 

9b. Child Care Calendar. A calendar format is used in Section DD to note the months since 
random assignment when each type of care was used. The calendar also notes the months during 
this period when the mother was employed. Section DD also asks the child’s age when he/she 
first cared for him/herself and the number of hours the child cared for him/herself in the last 
week. For the primary child care arrangement for the focal child in the last week, there are 
questions about hours/week, group size, and number of care givers. The section ends with a 
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series of questions about reactions to the major child care arrangement. Gallup-Black gave a 
presentation on how the child care calendar is being administered in MFIP using CAPI. 
Participants were able to examine an example of a completed child care calendar. 

The group discussed whether school age children experienced enough child care beyond school 
hours to warrant the use of a child care calendar format, given the time that this format requires 
to complete. Gallup-Black noted that in the interviews completed thus far in MFIP, quite a bit of 
child care is being reported for the children in the sample, who are school-age children. Zaslow 
noted that there is evidence linking the development of school-age children to the quality of the 
before and after school programs they participate in. It was noted that child care for school age 
children was a topic in the recent White House Conference on child care. The group decided to 
retain the child care calendar. 

While the group favored retaining the child care calendar, the issue remained regarding the 
period of time it should cover. In some of the state evaluations a period of more than two years 
would have elapsed since random assignment. The decision was made to focus the child care 
calendar on the two years prior to the survey, with the option of going back further, particularly 
for children between the ages of 5 to 8 years old at the time of the survey. 

Zaslow noted that the MFIP items regarding child care can be used to get a measure of stability 
of child care, but that it should be considered a proxy measure rather than an exact measure of 
. child care stability. The MFIP questions ask about the number of different providers or 
arrangements of each type used since random assignment. Yet, if a mother stopped using a care 
situation but then resumed using it, this will not be accounted for in the tally. The calendar notes 
months during which particular types of care were used (and types of care used concurrently), but 
does not note start and stop dates for specific arrangements. The group decided that the coverage 
of child care stability was sufficient given the approach used in MFIP. 

It was noted that it will be critical for all of the states to be using the same definitions about each 
child care type. Child Trends staff will provide all the states with MFIP’S question by question 
specifications about types of care. 

10. HOME ENVIRONMENT AND PARENTING PRACTICES 

Sharon McGroder led the discussion on this set of constructs. Four aspects of the home 
environment were proposed as common core constructs for mixed-mode surveys, with a fifth 
core construct if surveys are to be conducted in the home: child abuse and neglect; domestic 
violence and abusive relationships; family routines; mothers’ feelings of aggravation; and 
characteristics of the home environment. 

In general, participation in welfare-to-work activities may influence many aspects of the clients’ 
home life, and these changes are likely to affect children. Some policy makers fear that the 
added stressor of having to participate in welfare-to-work programs may be so great as to lead 
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some parents to neglect and/or abuse their children. There is a strong link between low income 
and family violence, and there is growing evidence of domestic violence suffered by participants 
in welfare reform programs. Mothers who are mandated to participate in employment and 
educational activities might also experience an increase or a decrease in the degree to which their 
lives are routinized, or follow a regular schedule. For example, mandated employment may force 
mothers to establish regular routines, such as regular bedtimes and mealtimes. Children can 
benefit from such predictability. Alternatively, the home environment may become more 
disorganized and less routinized as a result of increased stress or because families have 
difficulties managing employment, transportation, and child care arrangements, and children may 
suffer from the increased disorganization. 

Mothers who are mandated to participate in employment and educational activities might 
experience on the one hand, an increase in their subjective sense of stress, which can lead to 
more impatient and aggravated parenting practices. Also, questionable feelings of self-efficacy, 
time stress, or difficulties securing transportation or child care might increase mothers’ feelings 
of aggravation in the parenting role. On the other hand, involvement in activities beyond the 
family may diminish negative feelings toward the parenting role, with greater parenting 
aggravation associated with poorer child outcomes. 

10a. Child Abuse/Neglect. The MFIP questionnaire does not ask about child abuse and neglect 
because of the difficulty in obtaining accurate self-reports. The technical assistance team had 
recommended relying on administrative records on reported and substantiated abuse, which 
requires that states have the capacity to link their administrative records to survey members 
through a unique identifier. Several evaluators said that would be difficult to collect this 
administrative data with the limited dollars available. The costs vary by state and by whether the 
welfare office was already connected to the child welfare office data. They also wondered 
whether there was another way to collect this information besides administrative data. The 
quality of the data varies across states also, and in states where the data is of high quality, it could 
be a straightforward process. Howard Rolston then noted that several states provided estimates 
in their proposals. The prospectus sent to foundations reported that for the three states who 
proposed to collect this data administratively, the estimated total cost was $600,000 which 
includes administrative data on child support as well as child welfare. This issue will be 
revisited if additional funding becomes available. 

10b. Domestic violence/abusive relationships. Child Trends had proposed nine items asking 
whether anyone has made it difficult to find or keep a job for a variety of reasons, which were 
developed by MDRC and used in the National Evaluation of Welfare to Work Strategies Five 
Year survey. MFIP used these items verbatim (GGl-GG9b, pp. 91-93). Also, Child Trends had 
proposed items asking whether or not anyone had ever abused them in various ways (e.g., 
threatened with physical harm; actually hit or kicked), who, and how long ago (also from the 
National Evaluation of Welfare to Work Strategies Five Year survey). MFIP substituted one 
item, and structured the items to obtain even more detail than had been proposed (asking about 
who and how long ago for each of six types of abuse). In the interest of consistency and 



Child Trends, Inc. 



21 



December 9, 1997 Summary 



comprehensiveness of the measure, Child Trends recommended adopting the MFIP items and 
response category format (p.94-95). Iowa agreed. to field this measure and report on length. 

Martha Moorehouse asked how well these items were working so far in the field in MFIP.and in 
the National Evaluation of Welfare to Work Strategies Five Year survey. Adria Gallup-Black, 
who was heavily involved in designing as well as fielding these items, noted that these items are 
not asked in a phone survey because answering aloud may put the respondent in danger and does 
not yield honest answers. MFIP uses audio-CASI (Computer-Assisted Self-administered 
Interview) — which is the first time such abuse items are obtained from audio CASI — and 
preliminary checks indicate that these items are being answered. 

10c. Family Routines. Child Trends proposed a seven-item scale that captures the frequency 
with which families do such things as eat meals together, do household chores, and have regular 
bedtimes. MFIP uses a seven-item routines scale that drops one recommended item and adds 
another item (child eats breakfast at a regular time) (see page 69 of survey). Child Trends 
recommended asking all eight items and using the MFIP response categories, so that both the 
MFIP seven-item scale and the Child Trends-recommended seven-item scale can be created. No 
disagreements were noted with this recommendation. 

lOd. Aggravation/stress in parenting. Child Trends proposed a five-item aggravation in 
parenting scale. MFIP dropped one item, relying on a four-item scale (p. 97). They also used 
different response categories than had been initially proposed by Child Trends, though MFIP’s 
are consistent with what was used in the National Survey of America’s Families Study. Child 
Trends recommended that the five-item scale that was originally proposed be used by the other 
states in order to be comparable to data collected in other studies — including the National 
Evaluation of Welfare to Work Strategies Child Outcomes Study, the National Survey of 
American Families (Assessing New Federalism) and the Survey of Program Dynamics. Child 
Trends also recommended using MFIP’s response categories. The group approved this 
recommendation. 

lOe. Parenting Practices and the Home Environment. Involvement in welfare-to-work 
programs, as well as eventual employment, can influence many aspects of parenting, including 
involvement in joint activities, monitoring, discipline strategies, and the physical environment of 
the home. MFIP used the maternal report items from the HOME-Short Form (pages 70-74) as 
well as interviewer ratings of the home environment (page 1 04), as recommended by Child 
Trends in the revised measures notebook. In addition, because interviewer ratings of mother- 
child interactions from the HOME-SF require both the mother and child be in the home during 
the interview — which may require re-scheduling callbacks and, thus, is more expensive — 

MDRC consulted with Child Trends about options. In a prior memo, Kristin Moore reported on 
psychometric analyses showing what was lost by dropping these interviewer ratings and proposed 
obtaining additional maternal report items pertaining to discipline and monitoring in order to 
make up for the loss. MFIP includes these additional items as recommended by Child Trends. 
Child Trends recommended that the other states also use the HOME-SF (see pages 70-74 and 
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104) and additional items used in MFIP [see pages 97-98 (discipline), and 98-99 (monitoring)], 
to maintain comparability with MFIP. 

Finally, MFIP obtains interviewer ratings on physical characteristics of the neighborhood. (pages 
104-105 of the survey). Child Trends said that they would support other states’ decisions to 
collect this data, but because it is not in the common core, these items are optional. 

11. EDUCATION 

Tamara Halle led the discussion on educational outcomes. Improving family income may have a 
significant and positive influence on children’s educational outcomes. For example, increasing 
family income due to employment may make it possible for parents to provide their children with 
educational materials (such as books or toys) and the opportunity to engage in activities that 
could serve as sources of cognitive stimulation. Halle reviewed the five constructs related to 
educational outcomes included in the common core: engagement in school; attendance; 
performance; suspended or expelled; and grades. She noted that change in schools is covered in 
the stability/turbulence domain. 

11a. Engagement in school. Tamara Halle noted that MFIP included the four-item school 
engagement scale used in the National Survey of America’s Families that was recommended in 
the common core of constructs. All states agreed to used these same items. 

lib. School attendance. School attendance was measured in MFIP using questions from the 
New Hope survey (Section G, pages 51 & 52). These questions ask -- for each child in the family 
— what grade the child is in (with a code for “not in school” as an option), why the child is not 
attending school (including codes for hasn’t started school yet, is being home schooled, health 
reasons, parental choice, graduated, and dropped out), and what month and year the child last 
attended school. Halle noted that while the MFIP questions gather information on school 
absence or dropout, they do not capture much detailed information about children who are in 
school but who are experiencing problems with absenteeism or tardiness. Halle urged the states 
to consider including two of the recommended questions (Q29 & 31) from the Prospects survey 
in addition to the questions already in MFIP for this construct. The Prospect items ask how many 
days in the past four weeks the child was late to school or missed school. Some concern was 
voiced by the states that asking these questions for each child in the household would be too 
time-consuming. It was agreed that the questions would be asked for the focal child, and that 
states had the option of asking the question for all children, if time allowed. 

11c. School performance. MFIP included two items to measure school performance: one item 
(G1 1 , page 53) asks if the child has ever repeated a grade in school; the other (EE1 , page 86) 
asks whether the child ever received an academic honor. States agreed to ask these same 
questions. 
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Additionally, Halle pointed out that special education, ESL, and academically gifted 
classifications, or other educational classifications that may have a bearing .on school 
performance, could be obtained through school records if permission was received from the 
parent; parental permission for accessing school records appears on page 103 of the MFIP. 
survey. 

lid. Suspension/Expulsion. MFIP included one item from the New Hope survey (EE3, page 
86) to cover this construct. As specified in the core, this item covers any child residing in the 
household. States agreed to include this question in their surveys. 

lie. Grades. Information about grades is requested for all children in the household. MFIP 
includes a question on page 51 (item G6) that asks the parent to evaluate each child’s school 
performance, taking into account the child’s school work and report card grades. Halle pointed 
out that the MFIP question was an indirect measure of grades and stressed that the only way of 
getting reliable data on grades would be to access school records. It was agreed that, for 
consistency’s sake, that all states should use the MFIP question. However, states were urged to 
attempt to get parent permission to access school data; MFIP includes a form to obtain parental 
permission for school records on page 103 of the survey. 

12. HEALTH AND SAFETY 

Sharon McGroder led this discussion on children’s health and safety outcomes. She noted that 
mothers’ involvement in welfare-to-work activities may have implications for children’s health 
and safety, specifically: the occurrence of accidents and injuries; hunger and nutrition; overall 
health; and teen pregnancy. Mothers’ participation in welfare-to-work programs is likely to lead 
to increased time away from home. Unless the parent has a reliable child care arrangement for 
the child, the child may be at greater risk for accidents and injuries. Children who care for 
themselves or for their younger siblings may be placing themselves and their siblings at risk for 
accidents and injuries. Stress due to employment or other mandated activities may lead to more 
harsh discipline which, if extreme, could result in increased visits to the emergency room for 
accidents and injuries. Mothers’ time away from home without adequate alternative supervision 
of her children could lead to increased sexual activity and pregnancy among her adolescent 
children. Early childbearing and fathering of children is related to truncated educational and 
employment opportunities and could lead to welfare dependency among adolescent children of 
current recipients. On the other hand, work requirements, job training, time limits and other 
waiver provisions may increase adolescents’ motivation to delay parenthood. 

In terms of hunger and nutrition, if families experience an increase in income through working 
or a combination of work and benefits, they may be less likely than other recipients to experience 
food shortages. This may be especially true for the states that are experimenting with increased 
income disregards. On the other hand, sanctions for non-participation may reduce available 
income and thus availability of nutritious food. Children who do not receive adequate nutrition 
experience numerous physical problems as well as problems concentrating in school. 
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12a. Accidents, injuries, poisoning. Child Trends recommended a widely-used single item 
asking whether the respondent’s children had had an accident, injury, or poisoning requiring an 
emergency hospital visit. MFIP used this item verbatim (p. 60). Child Trends recommended that 
other states use this same item; there was agreement to do so. 

12b. General Health. Children’s general health may be affected by their mothers’ participation 
in welfare-to-work programs — through the mechanism described above. Child Trends 
recommended a widely-used single item pertaining to children’s general health. MFIP included 
this item (page 78). Child Trends recommended that other states use this same item; there was 
agreement to do so. 

12c. Teen Pregnancy. Child Trends recommended as part of the core that surveys ask which, 
if any, of the respondent’s children have ever had or fathered a baby, and how old they were. 
MFIP used these items with slightly improved wording (pages 86-87). Child Trends 
recommended that other states use the MFIP item; there was agreement to do so. 

12d. Hunger and Nutrition. Child Trends recommend the six-item USDA Food Security 
Module largely because this measure is being used in other national surveys which allows 
researchers to" “benchmark” their findings against this national sample. However, MFIP used 
three different items in their survey (page 63-64). Barbara Goldman noted that this was because 
these fewer items — approved by FNS — were used in the 12-month survey, and they kept them 
for consistency across waves. She also noted that these items came closer to the concepts 
relevant to those facing welfare time limits. Howard Rolston agreed that measuring food security 
was important, and wondered if the Department of Agriculture could help them get at hunger. 
Kristin Moore suggested, as a default, that other states use the six-item Food Security Module as 
recommended and Child Trends would follow up with USDA to verify that this is still the set of 
items that they would recommend. 

12e. Health at Birth. While mothers’ participation in welfare-to-work activities cannot affect 
whether or not the focal child was bom low-birth weight (under 5.5 pounds), received intensive 
care after birth, or was exposed prenatally to harmful substances, these aspects of the focal 
child’s health at birth can have ramifications for their health, development, and learning 
throughout childhood which, in turn, can also influence mothers’ successful participation in 
welfare-to-work activities. This suggests that analyses of program impacts on children’s 
outcomes would benefit from being able to control for the fact that some focal children were 
already “at risk” for poorer health outcomes before their mothers were exposed to program 
activities. Indeed, there may be differential program impacts on mothers and/or focal children 
depending on whether or not the focal child had early health problems. 

Sub-group measures, other than child gender, age and race/ethnicity, are difficult to obtain 
retrospectively. For example, child temperament might comprise a sub-group, but it was not 
possible to obtain temperament at birth and, while stable over time, it was too difficult and time- 
consuming to measure temperament among 5- to 12-year-olds. Health at birth, on the other hand, 
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is relatively easy to obtain with very few items asked retrospectively. The key is selecting items 
on which we are fairly confident that the mother can accurately report, especially for her older 
children. Because this construct was not discussed for the common core, we cannot recommend 
that states collect this information — we can only urge states to consider doing so. 

Intensive Perinatal care. For states interested in asking about intensive care after birth, they 
can ask: 

“Did (FOCAL CHILD) receive any newborn care in an intensive care unit, premature nursery, 
or any other type of special care unit?” (From NHIS, 1988) 

Yes 

No 

Low Birth weight. For states interested in asking about birth weight, they can ask: 

“How much did (FOCAL CHILD) weigh at birth? Was it . . . 

“Less than 3 1/4 pounds? 

“At least 3 1/4 but less than 5 Vi pounds? 

“Over 5 Vi pounds?” 

13. SOCIAL AND EMOTIONAL ADJUSTMENT 

Suzanne Le Menestrel led the discussion on the measurement of constructs in the social and 
emotional adjustment domain. There are several mechanisms through which welfare reform 
provisions might affect children’s social and emotional adjustment. Maternal stress or depression 
rates, which may increase because of reduced grants, sanctions, or time limits, or because of the 
mandatory nature of program participation, is related to an increased incidence of child behavior 
problems. Similarly, changes in family living arrangements or family income may affect 
children’s behavior. Alternatively, if welfare programs bring about enhanced maternal 
psychological well-being (for example, because of satisfaction with new employment roles), 
improved family economic well-being, and/or residence in a better neighborhood, behavior 
problems could diminish. Three constructs were selected by planning phase project participants 
to measure in the social and emotional adjustment domain. These include: behavior problems; 
arrests; and positive behaviors. 

13a. Behavior problems. MFIP has included the Behavior Problems Index (BPI). The BPI is a 
28-item parent report rating scale of children’s behaviors based on the Child Behavior Checklist 
developed by Thomas Achenbach. The BPI can distinguish between children who have received 
clinical treatment from those who have not, and taps some of the more common behavior 
syndromes in children, such as antisocial “acting out,” distractible-hyperactive behavior, and 
depressed-withdrawn behavior. It is widely used in many national level surveys (e.g., the 
NLSY79-CS) and has acceptable psychometric properties. Using the full BPI in the common 
core was approved by the state representatives and their evaluators. 
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13b. Arrests. This construct is being measured indirectly in MFIP by asking parents if their 
children have ever been in trouble with the police. Child Trends suggested adding a follow-up 
question about actual arrests because this will help sort out those children who are actually doing 
something wrong from those who may just be harassed by the police. Everyone agreed that this 
follow-up question should only be asked about children aged 10 and older. 

13c. Positive behaviors. These are being measured in MFIP with the 25-item Positive Behavior 
Scale (PBS). This scale was developed for use in the New Chance evaluation conducted by 
MDRC. The scale was designed to tap three aspects of positive behaviors including compliance 
and self-control; social competence; and autonomy. Child Trends suggested that states could 
include the shorter, seven-item social competence subscale rather than the full PBS. Project 
participants agreed to include the social competence subscale, with the full PBS as an option. 

V. INTERVIEWER TRAINING AND THE MONITORING OF DATA COLLECTION 
QUALITY 

1. QUESTIONNAIRES 

Lee Robeson, the survey director of the National Evaluation of Welfare to Work Strategies Five 
Year study from Response Analysis Corporation, talked about the importance of interviewer 
training and the monitoring of data collection quality. Robeson said that when data collection 
begins, there are two basic tools: the questionnaire and the interviewer. During training, the 
interviewers should come to understand everything that the survey developers understand. 
However, interviewers still need to be monitored in order to ensure that they are administering 
the questionnaires in the way in which the survey designers intended, and that they do not drift 
away from proper procedures over time. 

Robeson noted that interviewers are social, curious people, but they are not social scientists, 
child development experts, or standardized test administrators. They need clear instructions and 
definitions that are free from jargon. The instructions should be embedded in the questionnaire 
itself so that interviewers do not have to rely on manuals. The ideal questionnaire would require 
no training, instructions, or definitions. The closer the actual questionnaire comes to this ideal, 
the more likely that interviewers will be able to read the questionnaire in the way it was 
designed. The wording of the questionnaire should not be left to the interviewer to interpret. 
During training, the interviewer should feel comfortable in “trusting” the questionnaire so that he 
or she does not ad lib or feel that she or he has to ad lib. The most important tool that an 
interviewer needs in training is a manual with all of the necessary information. The training itself 
should provide multiple opportunities to practice the newly learned material. The interviewers 
also need to be prepared to deal with disadvantaged neighborhoods and households, abusive 
situations, and crime in neighborhoods and households. 

Following training, the quality of data being collected by questionnaires needs to be assessed. 
Robeson recommended asking interviewers to tape-record their interviews to better assess the 
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quality of the data being collected. The tapes can be used by supervisors to determine whether 
the interviewers are reading and recording the survey correctly and maintaining a good, 
comfortable interaction with the respondent. Monitoring should begin immediately. 

Interviewers want to do a good job and their needs can best be met by quick feedback. It is also 
important to let interviewers know at the start how and when their work will be evaluated. 
Robeson said that weekly, personal contact with interviewers is essential. 

There are several different sources that can be used to monitor the quality of questionnaires. 

First, the questionnaire instrument itself should be reviewed. Robeson suggested examining all 
key items and any items that might be potentially problematic. A review of questionnaires 
completed by the same interviewer to determine patterns is also helpful. Robeson also uses 
respondent tracking logs, notes, call logs, and site supervisors to monitor data collection quality. 
Respondents are also recontacted to verify that the interviews actually took place and were not 
forged. All monitoring efforts should also be documented to better track problems with 
interviewers. 

2. CALENDARS 

Please note that calendars were discussed more completely in the section on Child Care. Adria 
Gallup-Black reminded the group at this point that interviewers must receive very clear 
instructions in order to insure consistency in the data. The rules and guidelines for filling out the 
calendars should be explicit and practice time is necessary. 

3. INTERVIEWER OBSERVATIONS 

Martha Zaslow discussed the issue of training interviewers to make ratings. She noted that it has 
proven to be extremely helpful to have interviewer ratings in previous evaluation studies. These 
ratings give you the perspective of a different informant in the home. This permits you to discern 
whether a pattern of group differences, for example, is attributable entirely to maternal report or 
is corroborated by another informant. 

At the same time, asking interviewers to complete ratings requires them to make judgements. 

This in turn requires that special steps be taken during interviewer training. Steps that Child 
Trends recommends based on previous experience with interviewer ratings include: 

• Providing the interviewers with a sheet reminding them of the ratings they will need to 
complete and what aspects of the home environment they will need to be attending to during 
the home visit. This helps interviewers to be attentive to or look for the features of the 
environment that they will be rating. 

• Providing specific decision rules during training on how to complete the ratings. 
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• Building the question by question specifications for the interviewer ratings right into the form 
the interviewers will be completing. In this way, interviewers have the criteria for the rating 
available to them each time they make a rating. 

• Using words like “dark” and “not dark” rather than “yes” and “no” for ratings, to help to be 
certain that there is no problem in moving from the question or description to the fating. 

• Photographs can be used in training to support the introduction of the specific criteria for each 
of the ratings. However, photographs cannot cover every possible exemplar, and should be 
used to support a more extensive discussion of each of the ratings. 

Zaslow noted that at present, survey interviewers are not required to demonstrate inter-rater 
reliability on ratings of the home environment. This is a standard requirement in other types of 
research using ratings of the home environment. She noted that over time we should move in this 
direction. In the meantime it is critical to understand that training for the completion of 
interviewer ratings requires additional time and the provision of specific information on the 
criteria for each rating. 

In the group discussion, others confirmed the particular importance of allocating time to training 
for interviewer ratings. It was agreed that all of the states should receive question by question 
specifications from the MFIP survey on each of the ratings of the home environment. 

4. AUDIO COMPUTER ASSISTED SELF-ADMINISTERED INTERVIEWS (CASI) 

Adria Gallup-Black gave the group a quick demonstration of audio-CASI. This technology 
involves the use of a lap-top computer and headphones so that the respondent can answer 
sensitive questions (e.g., questions about domestic violence, behavioral problems, and 
depression) in private. One key to making audio-CASI a success is having a good tutorial for the 
respondent. The tutorial and the program should be user-friendly and easy to operate. Preloaded 
information also can make administration easier. For example, random assignment date and 
child’s name can be preloaded. As a respondent answers questions, her child’s name will appear 
on the screen (although the audio will say “your child”). 

Gallup-Black described three additional features of the audio-CASI. First, for very sensitive, 
items, respondents can choose to read the screen and turn off the sound, or she can turn off the 
screen and listen to the headphones. These features are helpful for working with respondents 
who can’t read. Second, respondents can go back and review or change previous answers. 
Finally, skip patterns can be programed in to simplify administration of the survey. 



Child Trends, Inc. 



29 



December 9, 1997 Summary 



VI. LESSONS LEARNED FROM THE NATIONAL EVALUATION OF WELFARE TO 
WORK STRATEGIES STUDY ABOUT INTERVIEWER TRAINING AND THE 
MONITORING OF DATA COLLECTION QUALITY 



Lee Robeson discussed the “lessons” that he has learned by fielding the National Evaluation of 
Welfare to Work Strategies (NEWS) surveys. He said that, given the lengthy field period, 
interviewer attrition and interviewer “drift” (e.g., interviewers asking questions differently than 
before) are two things that one must be prepared for. Initial over staffing helps to moderate 
attrition, although restaffing and retraining is often required. Monitoring interviewers’ 
performance using cassette tapes helps identify drift when it occurs. 

The second thing that he and his colleagues at Response Analysis have learned is that there is no 
way to adequately prepare interviewers to work in disadvantaged neighborhoods. Having older, 
more experienced interviewers is better, because they will often stay with the study after the 
training and first interview. Unfortunately, the older interviewers are more resistant to training, 
because they think they know how to do things better. 

Robeson has also learned that it is essential to include personal interaction in training and 
monitoring. Survey firms should insist on weekly contact with interviewers and should hold 
refresher training sessions every few months. These sessions help to build morale and boost 
production. These periodic meetings are particularly important with a lengthy data collection 
time. Validating the interviews is also important. The interviewers should be told that 100 
percent of their interviews are being validated. 

Finally, Robeson said that reliability is an important concern when relying on interviewers to 
make assessments, particularly of the home environment. Interviewers are not a homogeneous 
group; they come from dissimilar backgrounds. Without adequate training they will not all view 
a situation the same way. More often than not, they will not agree with the researcher’s view of 
the situation either. To ensure reliable results, thorough training is required. He offered the 
example of training interviewers to rate aspects of the physical environment. Showing 
photographs labeled with the appropriate rating is a beginning, but the training should go beyond 
that, for example, pointing out particular parts of the photograph that lead to a particular rating, 
and implementing training that has been shown to provide high inter-rater reliability. 

VII. LESSONS LEARNED FROM FIELDING THE MINNESOTA FAMILY 
INVESTMENT (MFIP) PROGRAM QUESTIONNAIRE 

Adria Gallup-Black from Manpower Demonstration Research Corporation discussed the lessons 
that she has learned from being involved in the fielding of the MFIP questionnaire. First, she 
said that tracking respondents has been a big issue, not necessarily because of refusals but 
because of frequent moves. This problem with tracking has cost implications for the study, and 
the interviewers have to be creative in finding the respondents. They have found that incentives 
are useful in keeping the refusal rate down (after people have been found in the first place), but 
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some respondents want more than $15 for the time and trouble of an intensive interview. Gallup- 

Black said that a questionnaire like MFIP’s cannot be fielded without incentives. 

VIII. TECHNICAL ASSISTANCE NEEDS FOR THE OPERATIONAL PHASE 

Kristin Moore led the discussion about the technical assistance needs of the state representatives 

and third-party evaluators. Below is the list of technical assistance needs that will be addressed: 

• Child Trends will consider how to create a ListServe so that project participants can share 
information about their evaluations. 

• Decide on which measures should be asked of respondents when the focal child is not living 
with the mother. 

• Distribute questionnaire instruments and interviewer training materials to project participants. 

• Child Trends will review drafts of questionnaires and provided feedback to the evaluators. 

• Consistent criteria across the five evaluations for selecting the focal child need to be 
developed. 
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Constructs in Italics = In-Home Survey 
All Child = All Child Module 
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